Assessing quality of service using digital watermark information

ABSTRACT

The disclosure details methods of measuring the quality of service of received media signals by analyzing digital watermarks embedded in such signals. The quality of a received video or audio signal can thereby be assessed without having the original version of the signal before transmission. Instead, the strength or quality of the embedded digital watermark is analyzed to determine the quality of the received signal. The degradation of a watermark signal is used to assess quality of service of signals, such as audio and video. Several other features and arrangements are also detailed.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a Continuation of U.S. application Ser. No.11/866,145, filed Oct. 2, 2007 (now U.S. Pat. No. 7,656,930), which is aContinuation of U.S. application Ser. No. 09/951,143, filed Sep. 10,2001 (now U.S. Pat. No. 7,277,468). U.S. patent application Ser. No.09/951,143 is a Continuation-In-Part of U.S. application Ser. No.09/938,870 (now U.S. Pat. No. 7,246,239), filed Aug. 23, 2001. U.S.patent application Ser. No. 09/938,870 is a Continuation-In-Part of U.S.application Ser. No. 09/840,016 (now U.S. Pat. No. 6,760,464), filedApr. 20, 2001. U.S. application Ser. No. 09/840,016 is aContinuation-In-Part of U.S. application Ser. No. 09/731,456 (now U.S.Pat. No. 7,346,776), filed Dec. 6, 2000. U.S. application Ser. No.09/731,456 claims priority from Provisional Application U.S. Application60/232,163, filed Sep. 11, 2000. The above patents and applications arehereby incorporated herein by reference.

TECHNICAL FIELD

The present technology relates to stenography, data hiding, andauthentication of media signals, such as images and audio signals.

BACKGROUND AND SUMMARY

Digital watermarking is a process for modifying physical or electronicmedia to embed a machine-readable code into the media. The media may bemodified such that the embedded code is imperceptible or nearlyimperceptible to the user, yet may be detected through an automateddetection process. Most commonly, digital watermarking is applied tomedia signals such as images, audio signals, and video signals. However,it may also be applied to other types of media objects, includingdocuments (e.g., through line, word or character shifting), software,multi-dimensional graphics models, and surface textures of objects.

Digital watermarking systems typically have two primary components: anencoder that embeds the watermark in a host media signal, and a decoderthat detects and reads the embedded watermark from a signal suspected ofcontaining a watermark (a suspect signal). The encoder embeds awatermark by altering the host media signal. The reading componentanalyzes a suspect signal to detect whether a watermark is present. Inapplications where the watermark encodes information, the readerextracts this information from the detected watermark.

Several particular watermarking techniques have been developed. Thereader is presumed to be familiar with the literature in this field.Particular techniques for embedding and detecting imperceptiblewatermarks in media signals are detailed in the assignee's applicationSer. No. 09/503,881 (now U.S. Pat. No. 6,614,914) and U.S. Pat. No.5,862,260, which are hereby incorporated by reference. Examples of otherwatermarking techniques are described in application Ser. No. 09/404,292(now U.S. Pat. No. 7,197,156), which is hereby incorporated byreference. Additional features of watermarks relating to authenticationof media signals and fragile watermarks are described in applications60/198,138, Ser. No. 09/498,223 (now U.S. Pat. No. 6,574,350), Ser. No.09/433,104 (now U.S. Pat. No. 6,636,615), and 60/232,163, which arehereby incorporated by reference.

The present technology provides a method of measuring the quality ofservice of media signals by analyzing digital watermarks embedded in areceived signal. This method enables the quality of the received videoor audio signal to be measured without having the original version ofthe signal before transmission. Instead, the method analyzes thestrength or quality of the embedded digital watermark to determine thequality of the received signal.

One aspect of the present technology is a method of measuring quality ofservice of a broadcast media signal using a digital watermark embeddedin the broadcast media signal. The method extracts a digital watermarkfrom the broadcast media signal, and evaluates the extracted digitalwatermark relative to a reference digital watermark to measuredegradation in quality of service of the broadcast media signal based ondifferences between the extracted and reference digital watermarks.

The method is implemented using fragile watermarks embedded in thebroadcast multimedia signal. These fragile watermarks, which areimperceptible in the broadcast signal, are based on digital watermarksused for authentication of media objects. One digital watermarkembedder, for example, transforms at least a portion of the media signalinto a set of frequency coefficients in a frequency domain. For example,it applies a Fast Fourier Transform (FFT) or other frequency transformto blocks of a media signal, such as an image, audio or video signal. Itadjusts a relationship between selected frequency coefficients to areference value. This adjustment is selected so that an alteration to bedetected, such as a re-sampling operation, lossy compression, broadcasttransmission, or digital to analog-analog to digital conversion, altersthe relationship. To detect the alteration, a detector computes therelationship in a potentially corrupted version of the signal.

Another digital watermark reader process evaluates signal peaks atselected frequency coefficients of the media signal. In a priorembedding process, the media signal has been modified to include peaksat the selected frequencies, such as by the technique summarized in theprevious paragraph. The method determines, based on degradation of thesignal peaks, whether the extent to which the quality of the mediasignal has been degraded. The frequency location of the peaks may varyfrom one application to the next. Another aspect of the presenttechnology is a watermark decoder, which includes a detector andanalyzer for determining alteration of a watermarked media signal. Thedetector correlates a calibration signal with a media signal suspectedof carrying a watermark to determine orientation parameters describingorientation of the media signal at embedding of the watermark. Thecalibration signal includes a set of peaks at selected frequencycoefficients. The analyzer orients the media signal using theorientation parameters and evaluates whether the media signal has beenaltered or degraded by examining signal peaks at selected frequencycoefficients in the media signal.

Further features will become apparent with reference to the followingdetailed description and accompanying drawings. The followingdescription details methods for using digital watermarks forauthenticating multimedia objects and measuring the quality of themultimedia objects as a function of digital watermark alteration. Italso describes alternative implementations and applications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram illustrating a process of embedding anauthentication watermark in a media signal.

FIG. 2 is a flow diagram illustrating a process of detecting theauthentication watermark from a potentially corrupted version of thewatermarked signal.

DETAILED DESCRIPTION

FIG. 1 is a flow diagram illustrating a process of embedding anauthentication watermark in an input media signal (100), and inparticular, in an image. The embedder begins by dividing a grayscaleimage into N×N blocks of samples at a specified resolution (102), whereN is a pre-defined integer. For each block, the embedder computes afrequency transform of the image samples in that block (104), namely, afast Fourier transform. From the mid-frequency and mid-high frequencycoefficients, the embedder selects M Fourier transform coefficients(106), where M is a pre-defined integer. The coefficient locations arefixed by a pre-defined pattern. For example, the locations are scatteredamong roughly 25 to 100 coefficient locations in the mid to mid-highfrequency range of a Fourier transform domain of a block of imagesamples where N ranges from 64 to 512 at spatial resolutions rangingfrom 75 to 600 dots per inch (DPI). The locations are symmetric aboutvertical and horizontal axes (and potentially diagonal axes) tofacilitate detection as explained further below.

For each of the M selected coefficients, x, the embedder computes aratio of the magnitude of a selected coefficient relative to themagnitude of its neighbors (108). In particular, it is a ratio of themagnitude of the selected coefficient to the average magnitude of thesurrounding neighbors:r(x)=Magnitude_of_(—) x/Average_of_Magnitude_of_Eight_Neighbors_of_(—) x

If r(x)<r, where r is a pre-defined reference value, the embedderincreases the magnitude of x such that:r(x)=r.

In this implementation, the value of r is a pre-defined constant. Thereference may be derived dynamically from the input media signal. Also,the reference may be selected from a table of values so as to select thevalue of r in the table at the minimum distance from r(x). Theadjustment to the host image is selected so as to be imperceptible orsubstantially imperceptible to a user in an output form of thewatermarked signal.

Next, the embedder computes the inverse fast Fourier transform on eachblock to obtain the watermarked grayscale image (112). The watermarkedimage (114) may then undergo one or more transformations, such asdigital to analog conversion, printing, scanning, analog to digitalconversion, photocopying, etc. These transformations tend to corrupt thewatermarked image in a predictable way.

The watermarking process of FIG. 1 may be combined with anotherwatermarking process to embed other watermarks, either robust or fragileto transformations such as sampling distortions, geometric distortions,scaling, rotation, cropping, etc. In particular, the process may becombined with an embedding process described in application Ser. No.09/503,881 (now U.S. Pat. No. 6,614,914) or U.S. Pat. No. 5,862,260 toencode a calibration signal that enables a detector to compensate fordistortions such as scaling, rotation, translation, differential scale,shear, etc. In one implementation, for example, the calibration signalcomprises an array of impulse or delta functions scattered in a patternin the Fourier domain of each block of image samples. To embed thepattern, the embedder perceptually adapts the calibration signal to thehost image block and adds it to that block. The impulse functions of thecalibration signal have a pre-defined magnitude and pseudo-random phase.To make the calibration signal less perceptible yet detectable, theembedder modulates the energy of the calibration signal according to thedata hiding attributes (e.g., local contrast) of the image samples towhich it is added. Preferably, the locations of the impulse functionsare scattered across a range of frequencies to make them robust totransformations like spatial scaling, rotation, scanning, printing, andlossy compression. Further, they are preferably arranged to be symmetricabout vertical and horizontal axes in the Fourier domain to facilitatedetection after flipping or rotating the watermarked image.

The frequency coefficient locations selected for the method illustratedin FIG. 1 may be mutually exclusive or overlap the coefficient locationsof the calibration signal. The calibration signal preferably has impulsefunctions at lower frequencies to survive compression, scanning,printing, etc. while the pattern of coefficients employed in FIG. 1includes coefficients at locations that are likely to be impacted byalterations to be detected, such as printing, scanning and photocopying.In the case where they overlap, the modification of the coefficientsaccording to FIG. 1 is implemented so as not to interfere with thecalibration signal. In particular, the embedder adjusts the selectedcoefficients as shown in FIG. 1 after the impulse functions of thecalibration signal have been introduced, or the embedder calculates thewatermarked signal taking into account the changes of the coefficientvalues due to the calibration signal and the process of FIG. 1.

Another approach is to adjust the selected frequency coefficients in themethod of FIG. 1 so that those coefficients act as both a calibrationsignal and an authentication signal. The locations of the coefficientsfor the method of FIG. 1 and the delta functions of the calibrationsignal are the same. The embedder increases the magnitudes of selectedmid to mid-high frequency coefficients relative to their neighbors toachieve the desired relationship with neighboring coefficients forauthentication purposes. Since this modulation includes the addition ofa delta function to the selected coefficients, it also inherently embedsa calibration signal comprised of delta functions at the selectedlocations. To compensate for rotation and scale, the detector performs aFourier Mellin transform of the suspect signal and the calibrationsignal into a log-polar space and then correlates the two signals. Thelocation of the correlation peak in log polar space provides the spatialscale and rotation parameters. These parameters may then be used tocompensate for rotation and scale changes before performing additionalwatermark decoding operations, such as the authentication operations ofFIG. 2.

To compute translation, the delta functions added to the selectedcoefficients may be given a known pseudorandom phase. In this case, thedetector correlates the phase information of the calibration signal withthe suspect signal after compensating for rotation and scale. Thelocation of the correlation peak gives the translation offset in thehorizontal and vertical directions.

In addition to being integrated with other watermark signal components,the process of FIG. 1 may be combined with a robust watermark embeddingprocess to carry a multi-bit message payload carrying metadata or a linkto metadata stored in an external database. Example implementations forembedding this type of robust watermark are described in applicationSer. No. 09/503,881 (now U.S. Pat. No. 6,614,914) and U.S. Pat. No.5,862,260.

FIG. 2 is a flow diagram illustrating a process of detecting theauthentication watermark from a potentially corrupted version of thewatermarked media signal (120) from the process of FIG. 1. The firstfour steps (122) are the same as shown in the embedder. For each block,the detector computes the average of r(x), where x is over all Mselected coefficients (124),R=Average_of_(—) r(x)

The detector computes the average of R over all blocks (126),AR=Average_of_(—) R

A related approach is to use a weighted average as follows. For eachblock, the detector computes a weighted average of r(x), where x is overall M selected coefficients (124),R=Sum_of_(weight_for_location_(—) x*r(x))

In this approach, the weights are fixed positive constant, independentof the image, with the weight sum equal to 1. For copy detectionapplications, the weight for each location is adapted for printers andprinting substrates used to produce original printed items.

The weighting factors are determined such that, for these printers andsubstrates, originals will be statistically optimally differentiatedfrom copies. Based on our experiments, the weights in higher frequencycomponents are usually higher. However the weights in the highestfrequency components are actually tuned lower, because some reproductiondevices (like photo copy machines) capture the highest frequencyreasonably well, and the first (original) printing process introducesnoise to the highest frequency components in the original printed items.

After obtaining the weighted average R for each block, the detectorcomputes the average of R over all blocks (126),AR=Average_of_(—) R″

To detect whether the watermarked signal has undergone alterations, thedetector compares the average of R with a pre-defined threshold (128).If AR>=T, where T is a pre-defined threshold, then the detectorclassifies it as original. If AR<T, then the detector classifies it as acopy.

Depending on the application, the detector may indicate the result (130)to a user through some user interface (e.g., visual display, audiooutput such as text to speech synthesis, etc.). The detector may alsoindicate the result (130) to another software process or device to takefurther action, such as communicating the event to a another device ordatabase for logging, recording tracer data about the user or device inwhich the alteration is detected, linking the detecting device to anetwork resource such as a web site at a specified URL that informs theuser about usage rules, licensing opportunities, etc.

To make the process robust to geometric distortion, the detectorincludes a pre-processing phase in which it correlates a calibrationsignal with the potentially corrupted watermarked signal as described inapplication Ser. No. 09/503,881 (now U.S. Pat. No. 6,614,914) or U.S.Pat. No. 5,862,260. Using a Fourier Mellin transform, the detector mapsboth the calibration signal and the received signal into a log polarcoordinate space and correlates the signals (e.g., using generalizedmatched filters) to calculate estimates of rotation and scale. Aftercompensating for rotation and scale, the detector uses the phaseinformation of the calibration signal to compute translation, e.g., theorigin or reference point for each block. Further correlation operationsmay be used to compute differential scale (e.g., the change in scale inthe horizontal and vertical directions after watermarking). Aftercompensating for geometric distortion, the detector executes the processof FIG. 2 to detect alteration in the selected frequency coefficientsmodified according to the method shown in FIG. 1.

While the present technology is illustrated with respect to a specificimplementation, it may be implemented in a variety of alternative ways.For example, the above example specifically refers to a grayscale image.This example may be adapted to other types of images including video andstill imagery, color and monochrome images, etc. For color images, theembedding and detecting operations may be performed on two or more colorchannels, including luminance, chrominance or some other color channels.The embedding and detecting operations may be applied to frequencycoefficients of alternative frequency transforms, such as DCT andwavelet, to name a few.

The embedding process shown in FIG. 1 may be performed on a portion ofthe host signal to create a watermark signal that is combined with thehost signal. For example, in one possible implementation, the embedderpre-filters the host signal to yield a high pass filtered signalincluding content at the mid and high frequency ranges impacted by thewatermark. The embedder makes the modification to this filtered signal,and then combines the resulting modified signal with the originalsignal.

The embedding and detecting processes may also be integrated intocompression and decompression operations. For example, the frequencydomain transform may be executed as part of a compression process, suchas JPEG, JPEG 2000 or MPEG, where blocks of the signal are transformedinto a frequency domain. Once converted to the frequency domain,frequency coefficients may be adjusted as described above.

The embedding and detecting operations apply to other media types,including audio media signals. In addition, the frequency domaincoefficients may be selected and adjusted to reference values to detectother types of signal alteration, such as lossy compression, digital toanalog and analog to digital conversion, downsampling and upsampling,etc.

Semi-Fragile Watermarks

A related watermarking approach is to use an array of Fourier magnitudeimpulse functions with random phase (a calibration signal, also referredto as a watermark synchronization or orientation signal) forsemi-fragile, and copy and copy-attack resistant watermarks.Semi-fragile refers to a watermark that degrades in response to sometypes of degradation of the watermarked signal but not others. Inparticular for document authentication applications using such awatermark, the watermark decoder can determine if the watermark has beenscanned and printed or battered by normal usage, potentially while beingread with a web camera. The copy-attack relates to the assertion thatone can use noise-reduction, i.e. Wiener filters, to lift a watermarkand, then using threshold and masking techniques, one can re-embed it ina new image. Interestingly, these concepts are related because they bothinclude an additional scanning and printing cycle, assuming the copyattack works on printed, not only digital, content. This type ofsemi-fragile watermark can be used to determine if a watermarkeddocument has been copied, possibly using a high quality copier and lowquality reader, and as such, can stop copying and can be used to measurequality of service.

One approach to implementing a semi-fragile watermark is to embed extrasignal peaks in the Fourier magnitude domain that are of varyingintensity, and have the watermark decoder determine if the watermark hasbeen scanned and printed by the relative power of the extra and originalcalibration signal peaks. The extra peaks refer to a set of peaks usedto implement the semifragile watermark. The original calibration signalpeaks refer to the ones already included in the watermark to determineits orientation in a geometrically distorted version of the watermarkedsignal. For an example of such a calibration signal, see U.S. Pat. No.5,862,260 and application Ser. No. 09/503,881 (now U.S. Pat. No.6,614,914), which are incorporated by reference. Some peaks are referredto as “extra” because they are included in addition to other peaks thatform the original set of peaks in a calibration signal.

One advantage of including the semi-fragile watermark in the calibrationsignal is that the robust part of the watermark, which includes thedesired detailed information such as a unique ID, is the multi-bitmessage, whereas the fragile part, which is only used to determinecopying, is a few bit message. The fragile watermark can be consideredas a single bit (copied or not) but actually allows more information bybeing frequency specific, as described below. Interestingly andpotentially advantageously, the semi-fragile watermark is separate butinherently related to the robust watermark—thus they cannot be separatedfor successful copy attacks.

Specifically, the extra calibration signal peaks should be located atfrequencies that best discriminates between the printing and scanningprocess, normal scuffing and a web camera reader. These locations can bedetermined by analyzing the frequency response of printing, scanning,scuffing and web cameras for frequency differences.

For example, a printing-scanning process may represent high-frequenciesbetter than a camera, but not low frequencies. In addition, scuffing mayshow low-and-high frequency losses. Thus, the reader will be able todetermine if the watermark has been copied, involving an additionalscanning-printing process, by the relative intensities of the extra andoriginal calibration signal peaks at low and high frequencies. In thisexample, high-and-low frequency loss is acceptable, whereas only lowfrequency loss represents a copied watermark.

In addition, the extra calibration signal peaks could also be dependentupon the content of the host signal, thus providing additional defenseagainst the copy attack. For example, the host image samples could bebroken in 16 equal sub-blocks, and the location of the extra peaksdepends upon the average intensity of each quadrant to the total averageintensity. Or, if only a section of the image is visible to the reader,each 32 by 32 sample block could be used in the above calculationinstead of the complete image. Any “hash” of the host image thatsurvives a web camera reader (referred to as a perceptual hash) could beused. To this end, if the watermark is moved to another picture, afterit is read, it is less likely that the extra calibration signal peaklocations are correct, not to mention that the less intense calibrationsignal points have been removed by the additional scanning-printingprocess.

Alternatively, in regards to the copy attack, the content dependentinformation could be used to slightly move the location of a feworiginal calibration signal peaks, as opposed to adding extracalibration signal peaks. This means that the image content isimplicitly in the calibration signal's jitter, and the copy attack isless likely to succeed unless the read and embedded images have the sameperceptual hash. On the one hand, this approach may reduce robustness ofthe robust message to scaling, rotation and translation. On the otherhand, no extra bits containing the output of the perceptual hash need tobe embedded in the robust message.

Based upon a different basic approach for stopping the copy attack, onecould create a 16-bit key from the perceptual hash described above (orsimilar key from any perceptual hash) and use it to encrypt (using RSAor DES) or scramble (using XOR) the payload and CRC bits beforeembedding them with an embedding protocol, which may include convolutionand/or repetition. This means that the reader can only correctly decryptor descramble the payload and CRC bits if the perceptual hash of theread image matches that of the embedded image. Thus, the copy attack isless likely to be successful without requiring extra bits to be includedto carry the output of the perceptual hash. This 16-bit key could useany method of feature based identification or vector creation, such aslisted in U.S. Pat. Nos. 4,677,466, 5,436,653, 5,612,729, 5,572,246,5,621,454, and 5,918,223, and PCT patent applications WO01/20483 andWO01/20609, which are hereby incorporated by reference.

Broadcast Monitoring and Quality of Service with a Watermark

When content is watermarked with a unique identifier (ID), any receiverwith a watermark detector can monitor what content is retrieved. Thecontent can be identified by name via resolving the ID in a secondarydatabase that contains at least IDs and related names, potentiallyincluding content owners who should be informed that the content wasdistributed. The assignee has several patent applications related tothis technology. See, for example, U.S. patent application Ser. No.09/571,422, filed May 15, 2000 (now U.S. Pat. No. 6,947,571), Ser. No.09/563,664, filed May 2, 2000 (now U.S. Pat. No. 6,505,160), and Ser.No. 09/574,726, filed May 18, 2000, which are incorporated herein byreference.

However, an interesting improvement is that the quality of the watermarkcan be measured and used to measure quality of service for thedistributor, who most likely is a broadcaster who wants to know that itsbroadcasts are being received with high-quality.

The quality of the watermark can be determined in many fashionsincluding using semi-fragile watermarks as described in this documentwith the application of copy resistance in mind. Measuring thedegradation of the watermark in the received media signal provides anindicator of quality of service.

For a packet distribution system, such as IP (Internet Protocol), aQuality of Service (QoS) method based upon semi-fragile watermarks isbetter than looking for dropped packets since it determines the effectof those packets on the video or audio. Many Internet video and audioplayers can re-create packets, and during times of slow scene changes,the quality may not be degraded badly. In addition, when the digitalwatermarks embedded in the packet stream have time segmented payloadsthat repeat at a defined or synchronized interval in the video or audio,the QoS of the video or audio can be measured over time by measuring thequality of the imperceptible digital watermark in the received video oraudio stream.

Measuring the Watermark Signal for Authentication and Quality of Service

There are multiple metrics for assessing watermark strength, includingthe degree of correlation between the reference watermark signal and thedetected watermark signal, and a measure of symbol errors in the rawmessage estimates of the watermark message payload. One way to measurethe symbol errors is to reconstruct the raw message sequence using thesame error correction coding process of the watermark embedder on thevalid message extracted from the watermark. This process yields, forexample, a string of 1000 binary symbols, which can be compared with thebinary symbols estimated at the output of the spread spectrumdemodulator. The stronger the agreement between the reconstructed anddetected message, the stronger the watermark signal.

To illustrate this method, it is useful to review how to embed thedigital watermark message signal imperceptibly in the host media signal.In the embedder, the embedded bit sequence is created by errorcorrection encoding a message payload, such as BCH coding, turbo coding,convolutional coding, Reed Solomon, etc. This embedded bit sequence isthen spread spectrum modulated with a carrier signal, such as apseudorandom sequence and embedded into the host media signal by subtlymodifying the signal (e.g., adding a binary antipodal watermark signalresulting from the spread spectrum modulation to spatial or frequencydomain samples of the host media signal).

Now, referring to the watermark detector, an approach for measuring thestrength of the watermark signal is as follows:

1. Use the message payload read from the watermark to re-create theoriginal embedded bit sequence (including redundantly encoded bits fromerror correction coding) used for the watermark.

2. Convert the original bit sequence so that a zero is represented by −1and a one is represented by 1.

3. Multiply (element-wise) the soft-valued bit sequence used to decodethe watermark by the sequence of step 2. In particular, the digitalwatermark reader produces a soft-valued bit sequence estimated fromspread spectrum demodulating the watermark signal, and supplies thesoft-valued sequence to the error correction decoder, such as a Viterbidecoder, which produces an error corrected message payload. Thesoft-valued sequence represents an estimate of the original, errorcorrection encoded bit sequence values along with a probability orconfidence value for each bit sequence value. The reader derives thesoft value by aggregating (e.g., summing) the estimates from demodulatedchips of the spread spectrum sequence used to encode that bit.4. Create one or more measures of watermark strength from the sequenceresulting in the previous step. One such measure is the sum of thesquares of the values in the sequence. Another measure is the square ofthe sum of the values in the sequence. Other measurements are possibleas well. For example, soft bits associated with high frequencycomponents of the watermark signal may be analyzed to get a strengthmeasure attributed to high frequency components. Such high frequenciesare likely to be more sensitive to degradation due to photocopying,digital to analog and analog to digital conversion, scanning andre-printing, broadcast process distortion, etc.5. Compare the strength measures to thresholds to decide if the suspectimage has been captured from an original or a copy of the printedobject. For print object authentication, the threshold is derived byevaluating the difference in measured watermark strength of copied vs.original media objects on the subject printer platform used to createthe original, and a variety of copiers, scanners and printers used tocreate copies. For quality of service measurement, the measurement ofwatermark signal strength at a receiver provides an indicator of videoor audio signal quality at the receiver.

This same technique of measuring symbol errors can be applied to two ormore different watermarks embedded at different spatial resolutions.Each of the watermarks may have the same or different message payloads.In the first case where the watermarks have the same message payloads,the message extracted from one of the watermarks may be used to measurebit errors in each of the other watermarks. For example, the messagepayload from a robust watermark embedded at a low spatial resolution maybe used to measure the bit errors from a less robust watermark at ahigher spatial resolution. If the watermarks carry different messagepayloads, then error coding, such as convolutional, Reed Solomon, orTurbo coding, and error detection bits, such as CRC bits, can be used ineach message payload to ensure that the message is accurately decodedbefore re-creating the original, embedded bit sequence.

Using two or more different watermarks enables a threshold to be setbased on the ratio of the signal strength of the watermarks relative toeach other. In particular, the signal strength of a first watermark at ahigh resolution (600-1200 dpi) is divided by the signal strength of asecond watermark at a lower resolution (75-100 dpi). In each case, thesignal strength is measured using a measure of symbol errors or someother measure (e.g., correlation measure).

If the measured strength exceeds a threshold, the detector deems thewatermark signal to be authentic and generates an authentication signal.This signal may be a simple binary value indicating whether or not theobject is authentic, or a more complex image signal indicating where biterrors were detected in the scanned image. For quality of servicemeasurement, the ratio of signal strength provides a measure of thequality of service.

The watermark and host signal can be particularly tailored to detectcopying by photo-duplication and printing/re-scanning of the printedobject. Likewise, the watermark signal can be tailored to detect videoquality degradation for quality of service measurements. This entailsembedding the watermark at particular spatial and/or temporalfrequencies/resolutions that are likely to generate message symbolerrors when the object is re-printed or broadcast. This detectionprocess has an additional advantage in that it enables automaticauthentication and/or quality of service measurement, it can be usedwith lower quality camera devices such as web cams and common imagescanners, and it allows the watermark to serve the functions ofdetermining authenticity as well as carrying a message payload usefulfor a variety of applications. For video quality of servicemeasurements, the detection process can take place in the same hardwareused to handle the video signal (assuming the video has a digitalrepresentation).

The message payload can include an identifier or index to a databasethat stores information about the object or a link to a network resource(e.g., a web page on the Internet). The payload may also include acovert trace identifier associated with a particular authentic item,batch of items, printer, or distributor. This enables a counterfeitobject, or authentic object that has been printed without authority tobe detected and traced to a particular source, such as its printer,distributor or batch number.

The payload may also carry printer characteristics or printer typeinformation that enables the watermark reader to adapt its detectionroutines to printer types that generated the authentic object. Forexample, the payload may carry an identifier that specifies the type ofprint process used to create the authentic image, and more specifically,the attributes of the halftone screen. With this information, the readercan check authenticity by determining whether features associated withthe halftone screen exist in the printed object. Similarly, the readercan check for halftone screen attributes that indicate that a differenthalftone screen process has been used (e.g., a counterfeit has beencreated using a different halftone screen). One specific example is apayload that identifies the halftone screen type and paper type. Thereader extracts this payload from a robust watermark payload and thenanalyzes the halftone screen and paper attributes to see if they matchthe halftone type and paper type indicated in the watermark payload. Forexample, the halftone type can specify the type of unstable screen usedto create an authentic image. If this unstable screen is not detected(e.g., by absence of a watermark embedded in the unstable screen), thenthe image is considered to be a fake.

A related approach for analyzing halftone type is to look for halftoneattributes, like tell-tale signs of stochastic halftone screens vs.ordered dither matrix type screens. Dither matrix screens used in lowend printers tend to generate tell tale patterns, such as a pattern ofpeaks in the Fourier domain that differentiate the halftone process froma stochastic screen, such as an error diffusion process, which does notgenerate such tell-tale peaks. If the reader finds peaks where none wereanticipated, then the image is deemed a fake. Likewise, if the readerfinds no peaks where peaks were anticipated, then the image is alsodeemed a fake. Before performing such analysis, it is preferable to usethe embedded digital watermark to re-align the image to its originalorientation at the time of printing. Attributes due to the halftonescreen can then be evaluated in a proper spatial frame of reference. Forexample, if the original ordered dither matrix printer created an arrayof peaks in the Fourier domain, then the peak locations can be checkedmore accurately after the image is realigned.

For quality of service measurement of broadcast signals, the payload maybe used to carry information about the type of broadcast, or type ofvideo processing used to create the broadcast video. The detector canthen use this information to adapt the watermark signal measurements forthe type of broadcast or video processing environment. For example, forcertain types of broadcasts, watermark signal measurement can be made atselected frequencies and/or particular locations within the broadcastdata stream. Also, the payload can be used to trigger certain types ofquality measurements on surrounding frames of video from which thepayload was extracted, and/or on particular parts of the frame where thewatermark has been specifically embedded for quality of servicemeasurements.

The above methods for measuring quality of service of video and audiobroadcasts apply to both radio frequency broadcasts as well as digitalnetwork broadcasts, just to name a few examples. In the case of adigital signal, the quality of the received “raw” digital signal can bejudged by any number of Channel State Measurement techniques that havebeen proposed. In the context of multimedia transmitted digitally over anetwork (like the internet), there can be congestion and packet losses.In this case, the communication channel does not have a guaranteedbandwidth; it only has some statistical description of availability. Forvideo and audio, the solution is to use buffers at the receiver andtransmitter to even out the statistical fluctuations in bandwidth.Still, there may be temporary periods with frame dropouts and/or otherdistortion artifacts. In these cases, quality of service monitoring isused to determine the quality of the reception over the network. Thereceiver can measure quality by determining when frames of video oraudio have been lost or delayed. In addition, digital watermarksembedded in the video and/or audio can be used to give a more accuratemeasure of the actual quality of the delivered video; additionally, ithas the advantage that it is independent of the video/audio codingstandard used. In the case of quality of service monitoring on networks,the digital watermarks are preferably embedded temporally, as well asspatially (for media signals with a spatial component like video).

The digital watermark is embedded temporally by embedding it across timesegments, such as by spreading and/or repeating the watermark signalacross multiple frames, so that the watermark detector can assess thedegradation of the watermark over those time frames. For instance, thewatermark can be spread over time just as it is spread over space byspread spectrum modulating the watermark message with a carrier signalthat spans a particular sequence of time frames. The message can then berepeated over blocks of these time frames. The watermark may also carrya time dependent payload so that time frames where the video or audiosignal has been degraded can be identified through the payload. Forexample, portions of the stream where a watermark payload cannot bedecoded indicate portions of the stream where the quality of service hasbeen degraded.

CONCLUDING REMARKS

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, applicants incorporate by reference the patents andpatent applications referenced above.

The methods, processes, and systems described above may be implementedin hardware, software or a combination of hardware and software. Forexample, the embedding processes may be implemented in a programmablecomputer or a special purpose digital circuit. Similarly, detectingprocesses may be implemented in software, firmware, hardware, orcombinations of software, firmware and hardware. The methods andprocesses described above may be implemented in programs executed from asystem's memory (a computer readable medium, such as an electronic,optical or magnetic storage device).

The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theincorporated-by-reference patents/applications are also contemplated.

We claim:
 1. A method comprising: receiving a media signal transmittedover a communication channel from a transmitter, wherein the channelfrom the transmitter is susceptible to distortion that varies as afunction of time, causing degradation in quality of service, and whereinthe received media signal represents audio and/or video information;recreating a reference digital watermark from the received media signal;extracting a second digital watermark, using a processor, from thereceived media signal, wherein extraction comprises decoding pluralmessage symbols steganographically encoded as the second digitalwatermark in the received media signal; and evaluating the seconddigital watermark relative to the reference digital watermark to measuretime-based degradation in quality of service of the received mediasignal based on differences between the second and reference digitalwatermarks, wherein evaluating the second digital watermark comprisescomparing the decoded symbols with a set of reference symbols toidentify errors there between, and wherein the errors are indicative ofthe quality of service of the received media signal.
 2. The method ofclaim 1, wherein the received media signal represents video information,and wherein the second digital watermark extracted from the receivedmedia signal is spread over time and spread over space in the receivedmedia signal.
 3. The method of claim 1, wherein the plural messagesymbols of the second watermark message are spread spectrum demodulatedfrom the received media signal.
 4. The method of claim 1, wherein thereceived media signal was watermarked by modifying an original mediasignal to embed the second digital watermark therein, and whereinextracting comprises extracting the second digital watermark from thereceived media signal without using the original media signal.
 5. Themethod of claim 1, wherein the received media signal represents videoinformation, and wherein the second digital watermark is spread overtime and spread over space in the received media signal.
 6. A methodcomprising: receiving a media signal transmitted over a communicationchannel from a transmitter, wherein the channel from the transmitter issusceptible to distortion that varies over time, causing degradation inquality of service, and wherein the received media signal representsaudio and/or video information; recreating a reference digital watermarksignal from the received media signal; examining, using a processor, thereceived media signal for a steganographically encoded second digitalwatermark signal having components at plural spectral frequencies; andassessing the strength or quality of the second watermark signal byreference to the reference digital watermark signal to determinedegradation thereof, wherein the degradation of the second watermarksignal is indicative of the quality of service of the received mediasignal.
 7. The method of claim 6, wherein the received media signalcomprises an audio signal.
 8. The method of claim 6, further comprising;decoding plural bit auxiliary information from the steganographicallyencoded second digital watermark signal; and using the decodedinformation for a purpose unrelated to monitoring quality of service ofthe received media signal, wherein the second digital watermark signalserves at least a dual purpose.
 9. The method of claim 8, furthercomprising using the decoded information to identify the received mediasignal, or metadata related thereto.
 10. The method of claim 6, whereinthe second digital watermark signal conveys information that isdependent on the received media signal in which it is encoded.
 11. Themethod of claim 6, wherein the received media signal comprises videothat has been conveyed in MPEG form.
 12. The method of claim 6, whereinthe second digital watermark signal is encoded in frequency coefficientsof a wavelet domain representation of media content.
 13. The method ofclaim 6, wherein the received media signal represents video information,and wherein the second digital watermark signal is spread over time andspread over space in the received media signal.
 14. A method comprising:receiving a media signal transmitted over a communication channel from atransmitter, wherein the channel from the transmitter is susceptible todistortion that varies over time, causing degradation in quality ofservice, and wherein the media signal represents video information;decoding, using a processor, a digital watermark from the received mediasignal to obtain payload data, wherein the payload data adapts thequality of service assessment in accordance with a particular videoprocessing environment associated with the media signal; and usingcertain of the payload data to inform a particular quality of serviceassessment to be performed on the received media signal, wherein theparticular assessment performed on the signal is informed by dataconveyed by the signal.
 15. A method comprising: generating, using aprocessor, key data through a process that includes applying aperceptual hashing function to the content, wherein the contentcomprises audio or video information; using the key data to scramble orencrypt a message, wherein the message comprises a plural symbolpayload; and digitally watermarking the content with the scrambled orencrypted message, to yield digitally watermarked content.
 16. Anon-transitory computer-readable medium having instructions storedthereon, the instructions comprising: instructions to receive a mediasignal transmitted over a communication channel from a transmitter,wherein the channel from the transmitter is susceptible to distortionthat varies as a function of time, causing degradation in quality ofservice, wherein the media signal represents audio and/or videoinformation; instructions to recreate a reference digital watermark fromthe received media signal; instructions to extract a second digitalwatermark from the media signal, wherein extraction comprises decodingplural message symbols steganographically encoded as the second digitalwatermark in the media signal; and instructions to evaluate the seconddigital watermark relative to the reference digital watermark to measuretime-based degradation in quality of service of the media signal basedon differences between the second and reference digital watermarks,wherein evaluating the second digital watermark comprises comparing thedecoded symbols with a set of reference symbols to identify errors therebetween, and wherein the errors are indicative of the quality of serviceof the media signal.
 17. A non-transitory computer-readable mediumhaving instructions stored thereon, the instructions comprising:instructions to generate key data through a process that includesapplying a perceptual hashing function to the content, wherein thecontent comprises audio or video information; instructions to use thekey data to scramble or encrypt a message, wherein the message comprisesa plural symbol payload; and instructions to digitally watermark thecontent with the scrambled or encrypted message, to yield digitallywatermarked content.